PyViz Menu

01 Workflow Introduction

Let's say you want to make it easy to explore some dataset, i.e.:

  • Make a visualization of the data
  • Maybe add some custom widgets to see the effects of some variables
  • Then deploy the result as a web app.

You can definitely do that in Python, but you would expect to:

  • Spend days of effort to get some initial prototype working in a Jupyter notebook, every time
  • Work hard to tame the resulting opaque mishmash of domain-specific, widget, and plotting code
  • Start over nearly from scratch whenever you need to:
    • Deploy in a standalone server
    • Visualize different aspects of your data
    • Scale up to larger (>100K) datasets

Step-by-step data-science workflow

Here we'll show a simple, flexible, powerful, step-by-step workflow, explaining which open-source tools solve each of the problems involved:

  • Step 1: Get some data
  • Step 2: Prototype a plot in a notebook
  • Step 3: Define your domain model
  • Step 4: Get a widget-based UI for free
  • Step 5: Link your domain model to your visualization
  • Step 6: Widgets now control your interactive plots
  • Step 7: Deploy your dashboard
In [1]: 
import holoviews as hv, geoviews as gv, param, paramnb, parambokeh, dask.dataframe as dd

from colorcet import cm_n, fire
from bokeh.models import WMTSTileSource
from holoviews.operation import decimate
from holoviews.operation.datashader import datashade
from holoviews.streams import RangeXY
from cartopy import crs

Step 1: Get some data

  • Here we'll use a subset of the often-studied NYC Taxi dataset
  • About 12 million points of GPS locations from taxis
  • Stored in the efficient Parquet format for easy access
  • Loaded into a Dask dataframe for multi-core
    (and if needed, out-of-core or distributed) computation
In [2]: 
%time df = dd.read_parquet('../data/nyc_taxi_wide.parq').persist()
print(len(df))
df.head(2)

CPU times: user 4.66 s, sys: 1.66 s, total: 6.33 s
Wall time: 10.2 s
11842094
Out[2]:
tpep_pickup_datetime tpep_dropoff_datetime passenger_count trip_distance pickup_x pickup_y dropoff_x dropoff_y fare_amount tip_amount dropoff_hour pickup_hour
0 2015-01-15 19:05:39 2015-01-15 19:23:42 1 1.59 -8236963.0 4975552.5 -8234835.5 4975627.0 12.0 3.25 19 19
1 2015-01-10 20:33:38 2015-01-10 20:53:28 1 3.30 -8237826.0 4971752.5 -8237020.5 4976875.0 14.5 2.00 20 20

Step 2: Prototype a plot in a notebook

  • A text-based representation isn't very useful for big datasets like this, so we need to build a plot
  • But we don't want to start a software project, so we use HoloViews:
    • Simple, declarative way to annotate your data for visualization
    • Large library of Elements with associated visual representation
    • Elements combine (lay out or overlay) easily
  • And we'll want live interactivity, so we'll use a Bokeh plotting extension
  • Result:
In [3]: 
hv.extension('bokeh')
In [4]: 
points = hv.Points(df, ['pickup_x', 'pickup_y'])
decimate(points)
Out[4]:

Here Points declares an object wrapping df , visualized as a scatterplot of the pickup locations. decimate limits how many points will be sent to the browser so it won't crash.

As you can see, HoloViews makes it simple to pop up a visualization of your data, getting something on screen with only a few characters of typing. But it's not particularly pretty, so let's customize it a bit:

In [5]: 
options = dict(width=700, height=600, xaxis=None, yaxis=None, bgcolor='black')

decimate(points.opts(plot=options))
Out[5]:

That looks a bit better, but it's still decimating the data nearly beyond recognition, so let's try using Datashader to rasterize it into a fixed-size image to send to the browser:

In [6]: 
taxi_trips = datashade(points, cmap=fire).opts(plot=options)
taxi_trips
Out[6]:

Ok, that looks good now; there's clearly lots to explore in this dataset. To put it in context, let's overlay that on a map:

In [7]: 
url = 'https://server.arcgisonline.com/ArcGIS/rest/services/World_Imagery/MapServer/tile/{Z}/{Y}/{X}.jpg'
tiles = gv.WMTS(url, crs=crs.GOOGLE_MERCATOR)
taxi_trips = datashade(points, x_sampling=1, y_sampling=1, cmap=fire).opts(plot=options)
tiles * taxi_trips
Out[7]:

We could add lots more visual elements (laying out additional plots left and right, overlaying annotations, etc.), but let's say that this is our basic visualization we'll want to share.

To sum up what we've done so far, here are the complete 10 lines of code required to generate this geo-located interactive plot of millions of datapoints in Jupyter: 

import holoviews as hv, geoviews as gv, dask.dataframe as dd, cartopy.crs as crs
from colorcet import fire
from holoviews.operation.datashader import datashade

hv.extension('bokeh')
df = dd.read_parquet('../data/nyc_taxi_hours.parq/').persist()
options = dict(width=700, height=600, xaxis=None, yaxis=None, bgcolor='black')
points = hv.Points(df, ['pickup_x', 'pickup_y'])
taxi_trips = datashade(points, x_sampling=1, y_sampling=1, cmap=fire).opts(plot=options)
url = 'https://server.arcgisonline.com/ArcGIS/rest/services/World_Imagery/MapServer/tile/{Z}/{Y}/{X}.jpg'
tiles = gv.WMTS(url, crs=crs.GOOGLE_MERCATOR)
tiles * taxi_trips

Step 3: Define your domain model

Now that we've prototyped a nice plot, we could keep editing the code above to explore whatever aspects of the data we wished. However, often at this point we will want to start sharing our workflow with people who aren't familar with how to program visualizations in this way.

So the next step: figure out what we want our intended user to be able to change, and declare those variables or parameters with:

  • type and range checking
  • documentation strings
  • default values

The Param library allows declaring Python attributes having these features (and more, such as dynamic values and inheritance), letting you set up a well-defined space for a user (or you!) to explore.

NYC Taxi Parameters

In [8]: 
class NYCTaxiExplorer(hv.streams.Stream):
    alpha       = param.Magnitude(default=0.75, doc="Alpha value for the map opacity")
    plot        = param.ObjectSelector(default="pickup", objects=["pickup","dropoff"])
    colormap    = param.ObjectSelector(default=cm_n["fire"], objects=cm_n.values())
    passengers  = param.Range(default=(0, 10), bounds=(0, 10), doc="""
        Filter for taxi trips by number of passengers""")

Each Parameter is a normal Python attribute, but with special checks and functions run automatically when getting or setting.

Parameters capture your goals and your knowledge about your domain, declaratively.

Class level parameters

In [9]: 
NYCTaxiExplorer.alpha
Out[9]: 
0.75
In [10]: 
NYCTaxiExplorer.alpha = 0.5
NYCTaxiExplorer.alpha
Out[10]: 
0.5

Validation

In [11]: 
try:
   NYCTaxiExplorer.alpha = '0'
except Exception as e:
    print(e) 

Parameter 'alpha' only takes numeric values

Instance parameters

In [12]: 
explorer = NYCTaxiExplorer(alpha=0.6)
explorer.alpha
Out[12]: 
0.6
In [13]: 
NYCTaxiExplorer.alpha
Out[13]: 
0.5

Step 4: Get a widget-based UI for free

  • Parameters are purely declarative and independent of any widget toolkit, but contain all the information needed to build interactive widgets
  • ParamNB generates UIs in Jupyter from Parameters, using ipywidgets
In [14]: 
parambokeh.Widgets(NYCTaxiExplorer)
In [15]: 
NYCTaxiExplorer.passengers
Out[15]: 
(0, 10)
  • Declaration of parameters is independent of the UI library used
  • ParamBokeh generates UIs from the same Parameters, using Bokeh widgets, either in Jupyter or in Bokeh Server
In [16]: 
parambokeh.Widgets(NYCTaxiExplorer)

We've now defined the space that's available for exploration, and the next step is to link up the parameter space with the code that specifies the plot:

In [17]: 
class NYCTaxiExplorer(hv.streams.Stream):
    alpha       = param.Magnitude(default=0.75, doc="Alpha value for the map opacity")
    colormap    = param.ObjectSelector(default=cm_n["fire"], objects=cm_n.values())
    plot        = param.ObjectSelector(default="pickup",   objects=["pickup","dropoff"])
    passengers  = param.Range(default=(0, 10), bounds=(0, 10))

    def make_view(self, x_range=None, y_range=None, **kwargs):
        map_tiles = tiles.opts(style=dict(alpha=self.alpha), plot=options) 

        points = hv.Points(df, kdims=[self.plot+'_x', self.plot+'_y'], vdims=['passenger_count'])
        selected = points.select(passenger_count=self.passengers)
        taxi_trips = datashade(selected, x_sampling=1, y_sampling=1, cmap=self.colormap,
                               dynamic=False, x_range=x_range, y_range=y_range,
                               width=800, height=475)
        return map_tiles * taxi_trips

Note that the NYCTaxiExplorer class is entirely declarative (no widgets), and can be used "by hand" to provide range-checked and type-checked plotting for values from the declared parameter space:

In [18]: 
explorer = NYCTaxiExplorer(alpha=0.4, plot="dropoff")
explorer.make_view()
Out[18]:

Step 6: Widgets now control your interactive plots

But in practice, why not pop up the widgets to make it fully interactive?

In [19]: 
explorer = NYCTaxiExplorer()
paramnb.Widgets(explorer, callback=explorer.event)
hv.DynamicMap(explorer.make_view, streams=[explorer, RangeXY()])

Widget Javascript not detected.  It may not be installed or enabled properly.
Out[19]:
In [20]: 
explorer = NYCTaxiExplorer()
parambokeh.Widgets(explorer, callback=explorer.event)
hv.DynamicMap(explorer.make_view, streams=[explorer, RangeXY()])
Out[20]:

Step 7: Deploy your dashboard

Ok, now you've got something worth sharing, running inside Jupyter. But if you want to share your work with people who don't use Python, you'll now want to run a server with this same code.

  • If you used ParamBokeh , deploy with Bokeh Server :
    • Write the above code to a file
    • Add a declaration of which object should be served (see nyc_taxi/main.py below)
    • bokeh serve nyc_taxi/main.py

Complete dashboard code

In [21]: 
with open('../apps/nyc_taxi/main.py', 'r') as f:
    print(f.read())

import holoviews as hv, geoviews as gv, param, parambokeh, dask.dataframe as dd, cartopy.crs as crs

from colorcet import cm_n
from holoviews.operation.datashader import datashade
from holoviews.streams import RangeXY

hv.extension('bokeh', logo=False)

usecols = ['dropoff_x','dropoff_y','pickup_x','pickup_y','dropoff_hour','pickup_hour','passenger_count']
df = dd.read_parquet('../data/nyc_taxi_wide.parq')[usecols].persist()

url='https://server.arcgisonline.com/ArcGIS/rest/services/World_Imagery/MapServer/tile/{Z}/{Y}/{X}.jpg'
tiles = gv.WMTS(url,crs=crs.GOOGLE_MERCATOR)
options = dict(width=1000,height=600,xaxis=None,yaxis=None,bgcolor='black',show_grid=False)
max_pass = int(df.passenger_count.max().compute()+1)

class NYCTaxiExplorer(hv.streams.Stream):
    alpha      = param.Magnitude(default=0.75, doc="Alpha value for the map opacity")
    colormap   = param.ObjectSelector(default=cm_n["fire"], objects=cm_n.values())
    hour       = param.Integer(default=None, bounds=(0, 23), doc="All hours by default; drag to select one hour")
    passengers = param.Range(default=(0,max_pass), bounds=(0,max_pass))
    location   = param.ObjectSelector(default='dropoff', objects=['dropoff', 'pickup'])

    def make_view(self, x_range, y_range, **kwargs):
        map_tiles = tiles.opts(style=dict(alpha=self.alpha), plot=options)
        points = hv.Points(df, [self.location+'_x', self.location+'_y'], self.location+'_hour')
        selection = {self.location+"_hour":self.hour if self.hour else (0,24), "passenger_count":self.passengers}
        taxi_trips = datashade(points.select(**selection), x_sampling=1, y_sampling=1, cmap=self.colormap,
                               dynamic=False, x_range=x_range, y_range=y_range, width=1000, height=600)
        return map_tiles * taxi_trips

explorer = NYCTaxiExplorer(name="NYC Taxi Trips")
dmap = hv.DynamicMap(explorer.make_view, streams=[explorer, RangeXY()])

plot = hv.renderer('bokeh').instance(mode='server').get_plot(dmap)
parambokeh.Widgets(explorer, view_position='right', callback=explorer.event, plots=[plot.state],
                   mode='server')

Branching out

The other sections in this tutorial will expand on steps in this workflow, providing more step-by-step instructions for each of the major tasks. These techniques can create much more ambitious apps with very little additional code or effort:

Future work

  • Jupyter Dashboards Server not currently maintained; requires older ipywidgets version
  • Bokeh Server is mature and well supported, but does not currently support drag-and-drop layout like Jupyter Dashboards does
  • ParamBokeh and ParamNB still need some polishing and work to make them ready for widespread use
  • E.g. ParamNB and ParamBokeh should provide more flexible widget layouts
  • Lots of work to do!

Right click to download this notebook from GitHub.